Skip to content

Make Record pickle-safe (fix RecursionError via GCS IO manager)#85

Merged
jirhiker merged 1 commit into
mainfrom
fix/record-pickle-recursion
Jun 26, 2026
Merged

Make Record pickle-safe (fix RecursionError via GCS IO manager)#85
jirhiker merged 1 commit into
mainfrom
fix/record-pickle-recursion

Conversation

@jirhiker

Copy link
Copy Markdown
Member

Problem

Materializing a downstream asset (e.g. nm_waterlevel_trends) fails loading an upstream input:

RecursionError: maximum recursion depth exceeded
  backend/record.py, line 82, in __getattr__
    v = self._payload.get(attr)   [repeated 980+ times]

Record.__getattr__ reads self._payload. During unpickling (the GCS pickle IO manager passing data between assets), pickle probes getattr(obj, "__setstate__") before __dict__ is restored, so _payload doesn't exist yet → __getattr__ recurses on __getattr__('_payload') until it blows the stack. Triggers whenever a Record — or a payload dict nesting one — crosses between assets.

Fix

Guard __getattr__:

  • dunder lookups raise AttributeError (so pickle falls back to its default state restore);
  • _payload is read via self.__dict__.get("_payload"), which can't re-enter __getattr__.

Verification

  • Bare and nested Records now round-trip through pickle.dumps/loads.
  • Full backend suite passes (263 tests).

Note

This was committed to the #84 branch after that PR was squash-merged, so it didn't reach main. Standalone fix here — affects every IO-manager product (major-chemistry, waterlevel-trends, etc.), so worth merging promptly.

🤖 Generated with Claude Code

Record.__getattr__ read self._payload, so during unpickling — when pickle
probes getattr(obj, "__setstate__") before __dict__ is restored — it
recursed on the missing _payload until RecursionError. This broke the GCS
pickle IO manager whenever a Record (or a payload nesting one) crossed
between assets.

Guard __getattr__: dunder lookups raise AttributeError (so pickle falls
back to default state restore) and _payload is read via __dict__ to avoid
re-entry. Verified bare and nested Records round-trip through pickle; 263
tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
die-orchestration View in Cloud Jun 26, 2026 at 09:09 PM (UTC)

@jirhiker jirhiker merged commit 0195aea into main Jun 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant